eventservice: split large scanned transactions - #5511
Conversation
|
Skipping CI for Draft Pull Request. |
ti-chi-bot
Bot
added
do-not-merge/needs-linked-issue
release-note
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds row-level scan resumption and spill-backed large-transaction processing across eventstore and eventservice. It also adds progress metrics, cleanup paths, integration coverage, a dispatcher-reset failpoint, and a dependency declaration update. ChangesLarge Transaction Split & Spill
Dispatcher batch-reset failpoint
Dependency declaration
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant EventBroker
participant DispatcherStat
participant EventScanner
participant EventStore
participant SpillFile
EventBroker->>DispatcherStat: request scan cursor
DispatcherStat-->>EventBroker: range and resume position
EventBroker->>EventScanner: scan request
EventScanner->>EventStore: iterate with cursor
EventStore-->>EventScanner: event and scan position
EventScanner->>SpillFile: append large-transaction inserts
EventScanner-->>EventBroker: events and scan progress
EventBroker->>DispatcherStat: persist scan progress
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ti-chi-bot
Bot
added
the
size/XXL
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for row-level resume tokens and large transaction spilling in the event service. It adds a new EventIteratorWithScanPosition interface, updates the event store iterator to track and return opaque scan positions, and implements disk-based spilling for large transactions via largeTxnInsertSpill and largeTxnScanState. Review feedback identifies two key issues in large_txn_spill.go: a potential resource leak where a failed Close() in Cleanup() skips deleting the temporary file, and a loose record length check that could lead to OOM panics if the spill file is corrupted.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Signed-off-by: dongmen <414110582@qq.com>
# Conflicts: # pkg/eventservice/event_scanner.go
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Introduce pkg/spill as a shared temp-file record store with chunked append, handle reads, sequential reads, and cleanup. Refactor eventservice large transaction insert spill to reuse it while keeping RawKVEntry encoding local.
ti-chi-bot
Bot
added
do-not-merge/needs-triage-completed
and removed
do-not-merge/needs-linked-issue
labels
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
|
/test all |
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Document scan progress and transaction scan strategies, and simplify the spill insert drain control flow without changing its behavior. Signed-off-by: dongmen <414110582@qq.com>
asddongmen
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lidezhu, wk989898 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
lgtm
approved
and removed
needs-1-more-lgtm
Yes, thank you. I have added that part. |
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
|
/test pull-cdc-mysql-integration-heavy-next-gen |
Signed-off-by: dongmen <414110582@qq.com>
|
/test pull-cdc-mysql-integration-heavy-next-gen |
Signed-off-by: dongmen <414110582@qq.com>
|
/test pull-cdc-mysql-integration-heavy-next-gen |
|
/test all |
|
/retest |
asddongmen
removed
the
do-not-merge/hold
|
/retest |
1 similar comment
|
/retest |
3 participants
What problem does this PR solve?
Issue Number: close #5596
EventBroker scans a whole large transaction, which can OOM CDC.
What is changed and how it works?
This PR adds row-level EventStore resume positions so
transaction-atomicity=noneemits bounded fragments without advancing resolved-ts early. Deferred insert halves of unique-key updates are spilled to disk and drained or cleaned across reset/remove.TiUP E2E comparison
Runs used one CDC limited to 32 GiB, upstream/downstream TiDB, and the same workload.
masterThe 20 GiB OOM is caused mainly by realtime logpuller
txnMatcherPREWRITE caching and overlapping ingestion buffers, before EventBroker scan splitting applies. asddongmen/tigate#2 fixes this matcher pressure and passed a separate 20 GiB no-OOM test; whether to merge it will be evaluated separately.Check List
Tests
The integration-test suite covers all changed workflows: row resume, interruption, unique-key spill/drain, CMEK spill encryption, dispatcher reset, and end-to-end consistency. Focused unit tests cover edge cases.
Questions
Will it cause performance regression or break compatibility?
No compatibility break is expected. Splitting applies only when transaction atomicity permits it; unique-key updates may add local spill I/O.
Do you need to update user documentation, design documentation or monitoring documentation?
An internal design document is included; no user-facing update is required.
Release note